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ABSTRACT 



Standards-driven school reform is the national 



strategy to raise U.S. schools to a world-class level by the year 
2000. Schools and school districts must develop plans, that show how 
to integrate the sets of national ^of-Is, standards, and assessments. 
This paper describes a s choo 1 ~ improvement model (SIM) developed by 
the Gilbert (Iowa) Community Schools, National Computer Systems (NCS) 
Corporation, and the College of Education at Iow3 State University. 
The paper first reviews research on cognitive competence and 
outcomes-based education (OBE) . It th'-n applies cognitive theories 
and construct ivist strategies to learning and student assessment. 
Stages of the SIP program's review- and curriculum-development 
process are described. The final section describes the management 
information system that ties student outcomes to educational 
objectives and leads to curriculum revision. It is important that 
teachers have access to the management system. The management system 
benefits the school improvement process in that it communicates cleai 
expectations and progress toward those expectations to teachers, 
students, and parents. Two figures are included. (LMl) 
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Standards-Driven Curriculum Reform 
and a Computerized Assessment Process 

William Butler Yeats once said, "Education is not the filling of a pail, but 
the lighting of a fire." The GOALS 2000: Educate America Act will probably 
start some fires and fill some pails! 

Let's review our eight goals: 

• All children will arrive at school ready to learn. 

• The high school graduation rate will increase to at least 90 percent. 

• Students will master challenging subject matter. 

• Teachers will have access to training programs to improve their skills. 

• U.S. students will be first in the world in math and science. 

• All adult Americans will be. literate and able to compete in a global 
economy. 

• Every school will be free of drugs and violence. 

Every school will strive to increase parental involvement and 
participation in their children's education. (H.R. 1804, "GOALS 2000: 
Educate Amenta Act," March 21, 1994) 
I want to build upon a medical analogy today. When I was young, a 
doctor could check my brain by: 

feeling the bumps on my head, 

• looking in my eyes, ears, and throat with a flashlight, and 
checking my reflexes with a rubber hammer. 

Today we have CAT scans and magnetic resonance imaging! What we're 
doing in this seminar is an attempt, with the help of NCS, to move the 
technology for teachers from hammers and flashlights to MRI. 

Why do we need standards-driven reform? We've had two waves of 
reform - neither worked. The United States is moving toward becoming a 
nation of education haves and have-nots. This might be tolerable in England 
with its caste system, but in this country it is not only morally wrong — but it's 
tantamount to planting a social time bomb. 

In an increasingly competitive global society, one where information is 
power and where "learning is the new form of labor," we can't afford to 
stumble again. The first two waves of reform each took a pitfall! 

The reforms triggered by "A Nation at Risk" contained no new ideas. It 
was a definite stumbling. We were to have more required courses, more 
standardized tests, a longer school year, and more money for teachers. 

We are asking schools to prepare students, all students, for demanding, 
fast changing jobs of the future -- with the rigid structure and teaching 
methods designed for the factories of the early industrial age. 

What ve're about nationwide and here in this audience today is undoing 
the work of Fred Taylor and Henri Fayol. They proposed at the turn of the 
century a centralized, hierarchical style of management, with a rigid sense of 
time, and an accountability system based on loyalty to the system. We are 
still doing it today! 
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This was perhaps best personified by an anecdote told on my campus. A 
young art major told me of a class where they lectured to him about creativity 
and then measured his creativity with a true/false test. 

The paradigm has shifted from "Education is Teaching" to "Education is 
Learning." How many times you've heard people say, "I taught my kid to 
swim, but he still sinks to the bottom." That wasn't teaching, that was talking 
about it. 

Chester Finn and his colleagues in the U.S. Department of Education had 
a wonderful idea for wave three reform; set standards, deregulate schools and 
hold them accountable. 

Nationwide, I expect a continued deregulation of schools, but 
unfortunately, in many cases without accountability. We do have to redesign 
America's schools, but we can't have true freedom for schools without 
accountability. In fact, the movement of reform is a search for new ways of 
doing things that will make both students and teachers accountable. 
Graduates ought to be able to use their education in real-life situations. Rick 
Stiggins and Grant Wiggins have shown us some interesting ways to do that! 

Students aren't foolish. They know about winning carrots and avoiding 
sticks. What were trying to do is force students to take more responsibility for 
their own learning. Curricula must become more focused, because we're now 
asking about the performance of every student, not just the elite. 

Teachers are not likely to be willing to accept responsibility for student 
achievement unless they have some control over what's happening in the 
classroom. We need a culture of responsibility, with teachers who plan and 
evaluate their own work. 

What we're about to launch, both in this room and nationally, is a 
serious U.S. debate about curriculum. We've never really had a serious U.S. 
debate over curriculum, because we've allowed the norm-referenced test makers 
to conduct the debate on our behalf. 

Now, with the Educate America Act and the national goals and standards 
which are being produced, we now are establishing goals, deregulating the 
system, and hoping to produce new forms of accountability. 

This is perhaps very fitting -- the frontier is closed. We can't depend on 
our ecology any more as a nation, we now must live with what comes out of 
our heads, just as the Japanese do. 

Standards-driven school reform is the national strategy to raise U.S. 
schools to a world-class level by the year 2000. Standards for seven more core 
content areas will quickly follow the mathematics, social studies, and science 
standards presently available. These standards are not simple directions from 
the top; they must be translated at the state and local planning level into 
curriculum and instructional delivery systems. What we have is a 
monumental '.ask analysis job to do. 

Schools and school districts need a plan, a well-designed series of steps 
and examples, which shows how to integrate the sets of national goals, 
standards, and assessments being prepared by the various scholarly 
organizations into local planning, teaching, and assessment procedures They 



need a "local control" model which is an efficient guide that has been 
thoroughly field-tested for validity. We think the SIM approach is that model. 
A collaborative partnership including the Gilbert (Iowa) Community Schools, 
National Computer Systems (NCS) Corporation, and the College of Education at 
Iowa State University has been formed to develop and test such a model. The 
project is based upon 15 years of research by the School Improvement Model 
(SIM) research team at the ISU College of Education. 

At the national level, Richard Riley, Secretary of Education, and President 
Clinton intend to jump start GOALS 2000, the goals set by then-President 
George Bush and the nation's governors in 1989, by the first education bill of 
their administration, the "GOALS 2000: Educate America Act." 

The GOALS 2000: Educate America Act will tie federal dollars to goals as 
well as to the so-called "school delivery standards" that are to measure the 
conditions needed in schools to allow students to meet academic standards. 
You will hear a lot about "opportunity to learn" in the coming months. The 
proposed school delivery standards also include the specification of 
assessments and an accountability for reporting to the local public and the 
U.S. Department of Education. 

To make matters even more complicated, regardless of whether you have 
a GOALS 2000 grant, numerous groups are hard at work setting goals and 
standards. These groups also recommend means of assessment. The 
mathematics, science, geography, and civics reports are now available. During 
1994-95, the U.S. Department of Education is funding standards projects in the 
following areas: the arts, civics, English, foreign language, history, 
mathematics, and science. The National Council for Social Studies and the 
Nations! Council on Economics Education are working on their own to set 
standards in their disciplines. Imagine what it will be like when all of these 
goals and standards arrive in the mail! We all want world class schools, but 
how can we have them without swamping teachers and administrators in a sea 
of change? 

Each report, filled with goals, standards of performance, and suggested 
assessment techniques, will be dumped, one after another, on someone's desk. 
All of these reports target the year 2000 for implementation. What should be 
done? We must start with an understanding of the nature of cognitive ■ 
competence. 

COGNITIVE COMPETENCE 

There has been a revolution in the social sciences in the past twenty 
years. The revolution has been our knowledge of the nature of cognitive 
competence and the long path that leads to its attainment. This new 
knowledge tells us much about how we should create curricula, teach, and 
assess student performance. In psychology, the once firmly entrenched 
behaviorism of Thomdyke, Watson, Hull, Spensc, and Skinner has given way to 
several generations of mind as information processing system (Gardner, 1987). 

Research in cognitive and skills learning has dramatic importance to 
curriculum design, delivery, and assessment. Unfcrtunately, the religious 



fundamentalists who have attacked Outcome-Based Education have almost no 
knowledge of this revolution and practicing private school teachers have little 
more. 

Admittedly, implementation of modern views of learning in public 
schools is equally sketchy. David Lohman suggests three reasons for this slow 
progress despite enthusiastic acceptance by most professional educational 
organizations. (1) Many educators were well trained in methods based on 
behaviorism, particularly the theories of Skinner. Such well-entrenched beliefs 
are difficult to change, especially when the evidence which challenges the 
theory comes from texts, articles, and other printed materials. (2) Even 
teachers who were trained in recent years often have only the barest exposure 
to modern theories of thinking and its development. (3) Those who have been 
trained thoroughly in modern instructional psychology soon find they must 
work in an educational system built on the foundation of behaviorism 
(Lohman, 1993). 

Indeed, those of us who have taught for forty or more years will attest 
that many of the key features of the curriculum classroom organizations, and 
student evaluation have not changed much since we were trained in the 40's. 
Beliefs about the need for reinforcement, behavioral objectives, individualized 
instruction, and objective tests all are rooted in the work of Edward Thorndyke 
and his "Law of Effect." In his view, memory consisted of a vast collection of 
specific responses to specific stimuli. Transfer depended on whether two 
situations shared the same stimuli elements. Broad transfer was an 
improbable goal using these concepts. 

In cognitive theory, the organization of knowledge and transferability of 
skill are paramount. Most cognitive theories distinguish several different types 
of memory systems, different types of memory codes, and different types of 
mental processes that operate on the learning task. 

Many theorists distinguish between fact knowledge and skill knowledge. 
This basic dichotomy has several important implications for curriculum 
development and assessment. The critics of OBE from the religious right say 
they want a return to the basics and more rote learning. Unfortunately, the 
problem is not the learning of facts but the learning of thousands of 
disconnected facts. 

In spite of frequent claims to the contrary by some 
educators, much of education consists of an attempt to impart 
factual knowledge to students. Research in cognitive psychology 
tells us that the single most important thing we need to know 
about a student's factual knowledge is how richly and flexibly the 
student has organized this knowledge. Such organizational 
schemes show that the learner has distinguished main ideas from 
less important ideas from details, and has related this new 
knowledge to old knowledge .... 

Research also suggests that, contrary to much current 
educational practice, when learning factual knowledge, a good 
motto is 'less can be more.' Students in elementary biology are 



expected to learn thousands of technical, unfamiliar terms, to 
relate them to each other, and to apply this knowledge to 
problems in other domains. A thorough understanding of a 
smaller set of main ideas is much better than a vague and 
piecemeal understanding of a much larger set of ideas. (Lohman, 
1993) 

The spokespeople for the religious right who rail against the whole language 
approach to reading and insist upon much drill in phonics are clearly following 
behaviorist views from the past (Manatt, 1995). 

Learners usually achieve transfer only after much experience in applying 
newly acquired knowledge to an increasingly diverse array of problems. This 
means that teaching for transfer requires constant review within and between 
possible domains. Lohman argues that this is probably impossible to achieve 
unless teachers know what students are learning in other classes and content 
domains, and actively encourage them to discover relationships among 
domains. Compartmentalization of teaching leads to compartmentalization 
of knowledge. That has been what OBE experts have been saying all along. 

COGNITION, CURRICULUM, AND TESTING: FROM CORRELATIONS TO CAUSES 

John Bruer, in his delightful new book Schools for Thought , likes to use 
the medical doctor metaphor as I did earlier in this speech. 

Imagine that a paramedic squad has just rushed you to a hospital 
emergency room. You tell the attending physician you have severe chest 
pains, shortness of breath, and numbness in your left arm. The doctor notes 
these symptoms and examines you for others. He then says, "There is a 70 
percent chance you will be dead within 4 hours. You appear to have some 
general, life-threatening physical deficiency." On the basis of his clinical 
experiences with hundreds of other patients, the doctor may have made a 
highly reliable prediction. He derives his prediction from extensive knowledge 
about statistical correlations between symptoms and outcomes. 

Now imagine that is all the doctor could say or do . Imagine that the 
science underlying medical practice consisted entirely of unexplained 
statistical relations between symptoms and outcomes. If so, doctors could give 
prognoses, but could make no diagnoses. Therapies and preventive measures 
would, at best, be based on other unexplained correlations, on common sense, 
or on unfounded speculation. Fortunately, biological science supports medical 
practice. Biology helps explain symptoms in terms of internal, unobserved 
bodily processes. The doctor wouldn't say you had a general physical 
deficiency. He'd say you had a heart attack that damaged certain coronary 
tissues, and that this condition should respond to specific treatment. The 
doctor could even suggest some preventive actions, such as changing your diet 
and exercising, to help you avoid future coronary events. 

Many of our educational assessment tools, particularly standardized 
aptitude and achievement tests, rely on unexplained statistical correlations. 
This leaves the teacher-student relationship at the same level as the doctor - 
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patient relationship in the imaginary heart attack. Our tests, which are based 
on many other cases we have observed and on statistical relations among 
them, allow us to compare and rank students by their relative academic 
health and to make predictions about educational outcomes. They allow us to 
attribute unfavorable outcomes to general intellectual deficits, but they don't 
help us make educational diagnoses, begin informed therapy, or prescribe 
preventive measures. 

Because we have cognitive theories and have the idea of "constructivism" 
as a strategy, we can „ow support new approaches to student assessment. The 
study of cognition has attempted to explain observed differences in intelligent 
human performance in terms of internal mental mechanisms. This cognitive 
science is to education what biology is to medicine. Cognitive mechanisms 
under'ie and cause the statistical correlations we measure with standardized 
tests. If we know what causes the correlations, however, we can make 
diagnoses, prescribe appropriate learning therapy, and identify preventive 
measures. We can use tests not only to measure learning but also to improve 
it. 

Standardized test scores can't diagnose what might be wrong with 
teachers, schools, or the educational system. Falling test scores suggest that 
something may be amiss, but the scores themselves can't tell us what the 
problems are or how to fix them. 

Cognitive science offers a different theoretical basis for testing and 
evaluation, one that can provide diagnostic information to complement 
standardized, norm-referenced tests. Cognitive science offers a theory and 
methods to describe what is behind students' performances -- what is causing 
the symptoms -- within a subject domain. Cognitive theory attributes 
differences in performance to specific differences in mental representations and 
processes. Cognitive scientists might start out comparing experts and novices, 
but they also can, and do, describe the intervening levels of competence and 
how that competence develops. They offer us a developmental psychology of 
performance changes. 

If we can develop tests to trace changes in performance, we can trace 
students' learning trajectories in school subjects. We can describe their 
progress in terms of the representations they have and the mental processes 
they use. This is the information we need for educational diagnosis, therapy, 
and prevention. To underscore this different theoretical approach to testing, 
cognitive scientists sometimes call the new approach learning assessment 
rather than achievement testing. 

The cognitive theory of learning assessment is in its infancy. Widespread 
adoption and application will require much more work and is likely more 
than a few years away. 

Most instructional design efforts involve a minimum of four 
components; namely, a specification of (a) the goals to be met, (b) materials to 
be used, (c) teaching strategies to be employed, and (d) items and procedures 
for assessment. These components seem to be important for any domain of 
instruction imaginable. Specific curricula involve specific values for each of 
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the four components of instruction. Thus, curriculum designers often specify 
in great detail the goals, materials, teaching procedures, and assessments. The 
strength of such well-specified efforts is that they make a complete curriculum 
package that is relatively easy to implement and evaluate. There is also a 
potential problem with such efforts. The more complete the specification of 
the values for each instructional component, the less inclined teachers may be 
to map into the unique features of particular students and communities. 

CONSTRUCnVISM 

Constructivism is what we call the strategies developed by the cognitive 
revolution. Constructivism says that students design their own learning. They 
construct it. Constructivism is the basic philosophy driving the integration of 
curriculum in the '90s. 

To adopt constructivism almost requires an epiphany, a burning bush to 
convert a person trained in traditional instructional design systems. 

The traditional approach to schooling has reflected a view of knowledge 
as entities existing independent of the learner or any context. Since knowledge 
exists independently, understanding can be objective, absolute, and 
unconditional. Cognition, in this view, is regarded as the rule-based 
manipulation of symbols via processes that will ultimately be describable 
through the language of mathematics and logic. 

We are now beginning to see a change in the basic epistemology of 
schooling. A constructivist epistemology argues that knowledge is not acquired 
as a collection of abstract entities but rather is constructed in the context of 
the environment in which it is encountered. Context is integral to 
understanding; meaning arises from context and context is an integral part of 
that meaning. People construct knowledge socially, through collaboration and 
discussion. It is this social process that results in shared meaning and 
understanding. Find time to read Designing Environments for Constructive 
Learning by Duffy et al. 

The chapters in this book explore the implications of a constructivist 
view for the design of learning environments and for the role of technology in 
that design. An examination of these chapters makes it clear that 
constructivism is not a unified point of view. There seems to be general 
agreement as to the importance of the authenticity of the learning task and 
the context in which the student works. There is also general agreement on the 
importance of collaborative learning as a means of developing a richer 
understanding through considering alternative perspectives. 

Perhaps the greatest differences among cognitive scientists arise from 
focusing on constructivism as a learning theory or as a set of instructional 
strategies. As a learning theory, the focus of constructivism is on how we 
understand and what it means to understand. It is a lens through which we 
see the world: all learning is seen from a constructivist perspective. For even 
the most basic learning task, meaning is constructed and understanding occurs 
in context. There is no restriction on the instructional - strategies that can be 
used. If drill and practice is the most efficient strategy for developing the 
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particular ability and understanding, then that is the instructional strategy 
that should be used. The critical issue is that the drill and practice occur in a 
context of a larger problem so that the learner sees the learning activity as a 
means of developing a skill that is needed. The criterion for success is the 
ability of the learner to use that skill in the larger context. 

What we are proposing is not theory. The ideas Professor Stow and 
Principal Ashby will explain have been used by districts no more wealthy or 
sophisticated than your district. It doesn't require brilliant teachers or genius 
curriculum directors (although that would be nice!). This process means that 
in three short years you can have curriculum alignment in all subjects, pre- 
and post testing to get the diagnostic and accountability pay off, and 
formative assessment to assure opportunity to learn (OTL). 

DISTRICTS COMPLETING SIM 

Our original districts which served as the prototypes in SIM were 
Minneapolis, Edina, Northfield in Minnesota and Spirit Lake in Iowa. Breck 
School in Minneapolis was our independent school prototype. Since then the 
three year process has been used in Hot Springs County, Wyoming (with five 
years of achievement gains, Manatt and Holzman, 1991); Lincoln County, 
Wyoming; Monroe County, Florida; Valparaiso, Indiana; Gilbert Community 
Schools, Iowa; Maricopa County Accommodations Schools (Phoenix), Apache 
Junction, Cave Creek and Coolidge, all in Arizona. 

Now Professor Stow will describe the curriculum renewal process [usually 
a three year cycle: (1) Language Arts, Reading and Mathematics; (2) Social 
Studies and Science; and (3) Fine Arts and Practical Arts]. Then Dave Ashby 
will explain how computer platforms put precision into the operations. We 
looked at 20 platforms in 1993-94 (Woodward, 1994). Dave will describe our 
experiences with the two best platforms in our study. 

CURRICULUM RENEWAL 

Curriculum renewal is a process of updating the infrastructure of the 
district. Curriculum must constantly correspond to change. However, this 
does no seem to be happening, because one of the basic issues school districts 
face today is that curricula are not sufficiently focused on, organized for, or 
adapted to the realities in which they operate. In essence, they are out-of - 
date. 

During the curriculum review and development process time needs to be 
allocated for the representative K-12 committee(s) to meet, to discuss, and to 
reach consensus about issues within the content area being studied. Many 
tlements will shape their decisions. High quality curriculum materials will be 
a synthesis of these elements, state frameworks or guidelines, data from 
tests, research on teaching and learning, the district's educational goals, and 
standards from the national associations. 

By using this renewal process, the district will have a basis for all the 
decisions about curriculum and instructional practices. It will have a quality 
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control mechanism because the written/taught/tested curriculum will be 
aligned and will optimize learning for students. 

REVIEW PROCESS 

The review process is a critical task which is done prior to curriculum 
development. The committee reads and studies the literature, which includes 
selected reprints from journals and/or excerpts from texts. The committee 
members discuss the important ideas from each article or excerpt and generate 
a list. 

One set of information used at this point would be the standards, 
whenever they are available from the scholarly society. For example, the 
National Council of Teachers for Mathematics identified their 13 standards 
and published them in 1989. Many committees readily accept the standards 
because they express an answer to the question - "How good is good enough?" 
for the curriculum being developed for learners moving into the next century. 
There are other committees who do not want to use the standards because 
they were not locally-developed and, if used, would cause many changes in the 
curriculum and instructional practices within the school district. 

After a review of the literature has been done the committee discusses 
the current practices from Kindergarten through twelfth grade and generates a 
list of them. Next they are ready to compare the list from the literature with 
the current practices. This task identifies the "gaps". As the committee begins 
to develop the curriculum, these lists serve as reference points and very likely 
there will be "planned abandonment" occurring as they continue to develop 
the curriculum. 

CURRICULUM DEVELOPMENT (Exhibit A) 

A set of structured activities is used as the components of the 
Curriculum Development Framework are defined. 

A The philosophy or statement of beliefs about the content area is 
determined. Questions like "What beliefs about students should be 
used as a basis for curriculum development in this content area?"; 
"Why is this body of knowledge worthwhile?" will be discussed and a 
paragraph(s) will be written. This statement will be used as the 
foundation for future discussions. 

B. Several strands or themes are identified and defined for the K-12 
content area. These serve as categories or natural classifications and 
usually there are six or seven. Examples for Mathematics would be 
Operations (The Operations strand involves the study of addition, 
subtraction, multiplication, division.) and Numeration (The 
Numeration strand involves the study of number systems.). Example; 
in Language Arts would be Literature (The Literature strand promote 
the development of reading as a tool for learning and provides 
experiences with a variety of literary forms.) and Study Skills (The 
Study Skills strand promotes the systematic development of the 
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ability to acquire knowledge, organize information, and utilize 
thinking skills.). 

C Program Goals are the general intents within each strand. Typically, 
there are 2-3 per strand. These are not measurable, are visionary/ 
guiding stars, and are nonspecific. For Mathematics an example 
could be "To understand the relationship of numbers to each other," 
while in Language Arts one could use "To understand a variety of 
literary forms." 

D. The scope and sequence grids (Exhibit B) display the skills/ concepts 
taught within each program goal. The skills/concepts are generated 
through the brainstorming process. The grids are the "wheel that 
drives the whole process." They serve as a map for teachers to use 
while making their plans for various units of study. Examples of 
skills/concepts within the Mathematics program goal, "To understand 
the relationship of numbers to each other," could be ordinal numbers, 
place value, odd and even numbers, and expanded notation and in 
the Language Arts program goal, "To understand a variety of literary 
forms," they might be essay, short story, fiction and nonfiction, and 
basic story elements. 

Lots of discussion occurs while the committee members work on 
the grids. It is a give-and-take discussion. Close attention needs to be 
paid to the symbols and their definitions as they are placed on the 
grids. These symbols are (1) Introduce (I) = to teach formally for the 
first time, (2) Expand (E) = to build upon the skill/concept in a 
sequential manner, (3) Mastery (M) = to require demonstration of the 
use of a skill/concept correctly 80% of the time, and (4) Maintain 
Mastery (MM) = to require application of previously mastered 
skills/concepts w.--.h materials of increasing difficulty, or (5) Reinforce 
CR) = to provide instruction which will assist in maintaining mastery 
while utilizing previous learning. The \, E, M, and MM or R indicate 
the Level of Learning or the "extent" to which one teaches. 

E The component labeled Unit Plans consists of four subsets of 
information, Le^ (1) Learner Outcomes, (2) Evaluation Activities, 
(3) Teacher and Student Activities and (4) Instructional Resources. 
Only the first two will be discussed. 

(1) Each Learner Outcome (objective) is written by using the 
Elements of the Golden Triangle or (a) an observable behavior, 
(b) conditions of learning, and (c) criterion measure. Using the 
analogy of a triangle reinforces the use of these three elements 
when writing the statement which defines WHAT will be taught/ 
learned. The observable behavior states how the learner will 
perform when the learning is exhibited (writes, draws, charts, 
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maps). Conditions of learning provide the circumstances under 
which the behavior will be demonstrated (given a ruler; given two 
figures that represent fractions through twentieth; given a list of 
15 words). The criterion measure indicates what will be 
acceptable as evidence the learner has achieved the learner 
outcome (five characteristics; weighed to the nearest gram; 
correctly; four out of five). As the learner outcomes are written 
each shows a specified level of Bloom's Taxonomy and the Level 
of Learning. When writing learner outcomes be SMART (or 
specific; measurable; attainable; relevant; trackable). This 
acronym helps keep the committee members working on well - 
written statements. 

(2) The Criterion-Referenced Measures (CRM) Evaluation 
Activities (Assessment Techniques) belong in two categories - 
Traditional (or the paper-and-pencil type) and Nontraditional 
(or those which are performance-based, product, portfolio, or 
personal communication). The focus for this paper is on one of 
the traditional techniques, multiple-choice tests. This type of 
test is the most widely used objective test and the most versatile. 
These test items can measure all levels of Bloom's Taxonomy and 
a substantive amount of learning can be sampled in a relatively 
short time. 

To provide direction for the CRM Developers, test specifications need to 
be defined. This is a reduction process which displays the relative importance 
of all learner outcomes which have been written. This planned focus will move 
everyone in the same direction within the teaching/learning setting. 

Prior to writing CRMs a common language about testing, guidelines for 
writing stems and foils, the process for writing items, and materials needed to 
write test items are reviewed. 

The discussion which develops a common language centers around a 
definition for each of these (a) test item, (b) stem, (c) foil, (d) distractors, 
(e) validity, (f) reliability, (g) formative, and (h) summative. All of these terms 
play an important role when writing test items. 

Guidelines for CRM Developers to think about as they write stems 
include (a) focusing on a learner outcome, (b) providing a sample of ALL the 
learning, (c) using a question instead of a "blank" in the stem, (d) avoiding 
negatively worded stems, and (e) being concise along with being grammatically 
correct. Guidelines for writing foils stress (a) avoiding clues, (b) using four 
choices, (c) writing them grammatically correct, and (d) avoiding such choices 
as "none of the above", "all of the above", "not given", and "not enough 
information". 

The process for writing items includes beginning with the learner 
outcomes which have a Level of Learning labeled "mastery" and then moving 
to those with labels of introduce, expand, and maintain mastery (or reinforce). 
(At the lower grade levels there may not be anything at mastery so the writers 
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begin at another level.) When creating the stem on the Test Item Development 
Form, write the stem and the correct foil first. This should read as a complete 
thought. Next, write the three distractors which are the logical but wrong 
answers. Each distractor should be written after the CRM Developer has 
determined WHY it was chosen. 

Materials needed to write test items are the (1) test specifications, 
(2) test item development forms, (3) scope and sequence grids, and (4) learner 
outcomes for which the items are to be written. 

A recommendation for the number of items for the pre/posttest would be 
25 items K-l-2, 35 for 3-4-5, 45 for 6-7-8, and 60 for 9-12. The CRM Developers 
should write many more than those recommended because each test will be 
field-tested and some items will not withstand this process. Plus any "extras" 
can be placed in a bank of items so there will be alternatives from which to 
choose. 

Most tests will have some type of graphic(s) in them. Some of these are 
computer - generated while others are sketched by a graphic artist. It is always 
wise to ask "Does the art work contribute to helping the learner answer the 
item(s)?" 

Code numbers are used to connect all sections of the system together. 
Information found in the code includes subject area - grade level/course - 
strand - program goal - skill/concept - test item number matched to a specific 
learner outcome - Bloom's Taxonomy - Level of Learning. These details are 
used when reports showing test results are generated. 

Several drafts of the curriculum and tests are written over time in order 
to make them "just right." The well-defined five step "Writing Process" from 
the Bay Writing Project is discussed with committees as it relates to curriculum 
development. In this process Step One or Prewriting is the preparation time 
when ideas are generated, options are considered, and patterns are framed in 
the mind. These are used when the Philosophy, Strands, Program Goals, and 
Scope and Sequence Grids are developed. Step two, Drafting, is the early 
version of the learner outcomes and test items. Revising, step three, is the 
delicate reworking of the written materials by using a critiquing process. Step 
four, Editing, is adding, deleting, moving, combining, or substituting one 
learner outcome or test item for another. Publishing, step five, is when the 
materials are disseminated so they can be implemented. At this point a 
leadership cadre of teachers is ready to teach those teachers who will be 
implementing the materials, but were not involved in the developmental 
process. This cadre will teach others what the newly-developed curriculum 
and tests are all about. Our experiences have shown this is a very successful 
way to accomplish this task. 

In summary, the curriculum and criterion-referenced measures become 
the foundation for the teaching/learning process and serve as the district - 
specified blueprint. This user-friendly set of board-adopted materials defines 
affirmatively: "of all that could be taught/learned/tested, these are what are 
important in this district." Without such a plan, learning is left to chance and 
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worthwhile learning is achieved by accident or default, not by intention or 
design. 

PRECISION AND THE CURRICULUM 

Dr. Manatt and Dr. Stow have shown you a capsule of what it takes for 
proper curriculum design, including planning, articulation, assessment, 
selection and writing. What they suggest is a prerequisite to the next step. It is 
the next step that, unfortunately, rarely gets implemented. Too often the 
results of the curriculum development effort end up in a big thick notebook 
that sits on the curriculum coordinator's or the principal's shelf. Occasionally 
it finds its way to the teacher's shelf, but too many times, the adopted 
textbook series becomes the "real" curriculum guide instead of a resource. 

A great deal of effort goes into deciding what is to be taught as well , as 
the instructional process that follows the planning. Teachers spend a lot of 
time checking tests which they assume assess what was learned from what they 
taught. Sometimes the test items match the curriculum and instruction and 
sometimes they don't. To develop some kind of structured system for 
accountability, teachers will create logs, checklists, databases, or spreadsheets. 
However, that is a monstrous, time consuming job when doing it without the 
use of technology. This is where a computerized curriculum management 
system is needed. This does not mean that the contents of the big thick 
curriculum book are just typed into a computer that has a hard drive, because 
nothing is gained from that. 

There needs to be precision management of all the information relating 
to the planning, teaching, and assessment of the curriculum. A good 
management system provides that. It ties the outcomes (expectations), the 
objectives/goals, the knowledge strands, the individual lesson objectives, the 
learning hierarchy, and the assessment items together. Several loose ends may 
get tied together nicely with a management system. A good management 
system will not allow assessment items that have no objective tied to them. It 
will not allow objectives to be taught that are never evaluated. 

"It's what's up front that counts" could be a motto for the Gilbert 
Elementary School with regard to management systems. A partnership with 
Iowa State University, The NCS Corporation, and Gilbert Community Schools 
has provided an opportunity for evaluation and demonstration of an 
established management system called Performance Plus ®, and now we are 
implementing "ABACUS" which is NCS's acquisition to upgrade the quality of 
their management product line. 

The up-front work is the hardest and most time consuming. Developing 
the curriculum is aligning the goals and objectives to assessment items and 
takes a lot of time and work. But, once all this is in the system, the payback 
for the time and work starts. Quick and easy access to the developed 
curriculum, quick easy scoring of tests, pages of reports for parents, students, 
teachers, administrators and quick feedback for all concerned are the benefits. 
Comparison between students, item analysis of tests, progress reports which 
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have meaning (unlike the traditional letter grades) so parents can understand 
are all valuable tools. The quick results make it easy for teachers to design 
their reteaching loops while teachers without management systems are 
spending hours checking test papers by hand. 

Curriculum revision comes about as a result of the assessments. 
Assessment drives the curriculum. The curriculum becomes a living, evolving, 
accessible entity instead of a big thick book on a shelf. When you know what 
was planned, what was taught, and what was assessed, changes are much 
easier. Having a precise handle on what students know and don't know allows 
precision modification. 

In an ideal situation in a school, all teachers would have network access 
to the management system. This moves the big thick book to the teachers' 
desks. Teachers can easily get various reports on student progress or quick 
access to curriculum statements, objectives, and test item banks. The teachers 
would have access to a large selection of valid, reliable, test items. No need to 
worry about whether the students have the test, just print a new one that is 
just as valid and reliable with all new questions. Make-up tests and test 
retakes could be handled the same way. The superintendent smiles when, 
he/she receives the printed report of how students are progressing towards 
established outcomes and puts the information on the required state report. 

The basic components of a management system are student/ 
demographic information, course and teacher assignment, curriculum 
information, test development/item banks, and report. There is, of course, 
programming for scanning score sheets, including graphics on test items, 
printing of reports, and transferring data. Assessments are not limited to 
"bubble" sheets. Performance observation, building of portfolios of all kinds of 
information, and other measurement techniques should be built in. A good 
systems allows flexibility in report design to meet school needs. Data can be 
pulled out relating to equity issues, and other things that might bias 
performance. 

The start-up cost for the software for a management system can be 
expensive, but the layout is worthwhile. The cost is $5,000 or more for a small 
school. The cost ^. computers, networking, file servers, printers, and terminals 
comes in addition to the cost of the software. It may be necessary to have a 
clerical position to input data and score tests. A lot can be done for the price 
equal to that of a new school bus. 

Having a management system has been an awakening experience at 
Gilbert Elementary School. As teachers pulled objectives from their developed 
curriculum guide and began to fit them into the hierarchical structure in the 
management system they began to find out that what they had written down 
was not necessarily what they were teaching. They were forced, by the system, 
to restructure the goals and objectives to match what was really going on 
because the system forces the assessment items to match. Teachers began to 
split broad objectives/goals apart into more specific statements. In other 
words, when the teachers could actually see their curriculum structure, they 
found they needed to change it to fit their teaching. Of course, they can't 
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capriciously change their curriculum. The old established system (the 
curriculum coordinator, the principal, established curriculum development 
procedures, and the Board of Education policy book) prevents that. 

Teachers found that the textbook tests did not necessarily match what 
was taught and they became skeptical of the "canned" curriculum. They 
turned back to what had been designed by them. The teachers discovered test 
items on objectives that were not taught or tested. There are many high - 
quality published instructional materials and they are valuable resources when 
used appropriately. But, when it comes to deciding what needs to be taught, 
at least in Iowa, the local school district wants the authority. 

We are not denying that it takes a lot of time and work to get the 
curriculum into shape. Dr. Manatt and Dr. Stow do not visit a school for a 
couple of days and then disappear. They take on projects that last several 
years. But, when their work is done, and with the curriculum in a 
management system, the curriculum becomes easier to revise and the output 
from the system pays back for the time and work put in. A typical example 
would be: 

A Gilbert 6th grade science teacher asked for a test. The system 
operator hands the teacher the test within 15 minutes, 
including the scanforms, the answer key, and a list of the 
objectives to which each test item is tied. The test is given and 
the answer sheets are fed into the scanner. In another 15 
minutes, the tests are scored (for two class sections) and the 
results are handed to the teacher. The "results" package is 
delivered to the students and then to the parents. This packet 
includes the scanforms, the test booklet, a printout which shows 
how each student did on each test item and the percentage of 
mastery by objective. Also included is a class summary 
showing how the class or grade did by objective. This allows 
parents to compare their own student's performance to that of 
the class. In addition, the teacher gets a graphic printout of 
correct and incorrect answers by student and an item analysis 
of the test. At the end of the year or semester, the 
superintendent and curriculum coordinator receive a large 
printout of student progress toward all the objectives in all the 
subjects. It becomes very easy to spot the successes and the 
trouble spots. The results are quick. The only hazard would be 
if the results are treated like some treat the results of 
standardized tests. A pile of computer printouts goes on the 
shelf next to the dusty curriculum guide. The quality teacher 
will, at a glance, know what has been mastered and which 
students need to follow the reteaching loop and then acts on 
the information. 

The high school teachers will ask, "How can you send the test home? The 
students will have my test." We don't see that as a problem. A good test 
generator will mix up the answers and mix up the questions in several different 
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forms of the same test. It is also fairly easy to select a complete new test from 
a bank of many hundreds of proven reliable and valid items. If the students 
still perform very well on the test, then the teacher has a clue that perhaps 
time should be spent teaching other things because these specific objectives 
have been mastered. 

Recently, Gilbert Elementary teachers have asked for a printout of the 
objectives before they select the test or items for a test. When the building is 
networked, the teachers will do that via the terminal on their desk. 

Test item design is an area where Dr. Stow and Dr. Manatt spend a lot of 
time. The Gilbert Elementary teachers are learning about the diagnostic value 
of the foils that accompany the multiple choice items. Now, with ABACUS, 
performance observation and portfolio development are new areas we 
explored. "Bubble sheets" are not the only assessment tool. First graders don't 
do well on bubble sheets. 

A final comment about an unanticipated result of a computerized 
management system. This is quickly observable by an administrator and is 
now being noticed by the teachers. Something positive happens to teachers' 
confidence when they use such a system. They always thought they knew 
what they were doing. Now they can demonstrate it. Something positive 
happens to teachers when they realize that their curriculum design time was 
not a waste. The teachers taught what they planned to teach and the 
assessment tested what they taught. The results drive what happens next. 
"They have it learned or we go back and hit it again" is a comment heard over 
and over again. This is the essence qf criterion assessment. Standards are 
established and we work at it until we get there. Some students just take 
longer. There is no requirement that a certain percentage fail. The ownership 
goes back to the students. We are here to help them. They do the work. We 
help them learn. When teachers have this new confidence it transfers to 
students. The teachers know what they are doing. The students know what 
they need to learn. No guessing what the teachers want the students to 
remember. There are no tests that have "tricks" in them to be sure that no 
one gets all the answers correct. It is all up-front and the teachers and 
students attack it together. 

Businesses for many years have held themselves accountable for their 
products because it is a matter of survival. They spend the money and make 
the commitment to manage their production to keep themselves competitive. 
It is now becoming a matter of survival for schools to be able to precisely 
report to their constituents how the schools are doing with regard to teaching 
and learning. Many parents now have the option to pick a school of their 
choice via open enrollment. Private schools say that they can do it better. 
The media happily announces that the rest of the world is doing better in 
education. That is pressure to change. 

Is it possible that many schools really have improved, but lack the 
vehicle to communicate this improvement ".o their public? Parents assume 
letter grades mean something. We are happy to give them information that is 
more meaningful. Why not tell parents and students what is expected and 
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then give them detailed reports of student and school system progress towards 
those expectations? Sounds great, but manually, the handling of the data and 
information is impossible. A good management system is the tool. 
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